Dynamic local registration system and method

ABSTRACT

In accordance with the teachings described herein, systems and methods are provided for generating images for use in systems, e.g., imaging systems. The method includes receiving at least a first set of images, providing a first registration, providing a display, and displaying a first image on said display. Further, the method includes providing a user interface, providing a second registration, and displaying a second image in said user interface. Further, the systems include an image database, a display, and a registration engine. The registration engine includes software instructions stored in at least one memory device and executable by one or more processors.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.14/683,330, filed Apr. 10, 2015. Application Ser. No. 14/683,330 is adivisional of U.S. patent application Ser. No. 13/624,827, filed Sep.21, 2012, entitled “DYNAMIC LOCAL REGISTRATION SYSTEM AND METHOD”. Theentireties of the aforementioned applications are incorporated herein byreference.

TECHNICAL FIELD

The present invention relates generally to the field of image processingand more particularly to image registration.

BACKGROUND

Three dimensional medical scans of patients, such as CT (computedtomography), MR (magnetic resonance), US (ultra sound), or PET (positronemission tomography), produce a series of two dimensional (2D) imageslices that together make up 3D images. The two dimensional image slicescan be stored to create a set of primary or target images. A medicalprofessional may take another set of images of the patient that can bestored in memory to create a set of secondary or source images. Thesesecondary or source images may be compared with the primary or targetimages, for example, and processed through an image registration todefine a point to point correlation between the primary and thesecondary images. However, image registrations, particularly high orderregistrations including deformable registrations, of any volume may bedifficult to interpret and evaluate for accuracy, and doing so can betime consuming for medical professionals.

The aforementioned difficulty in understanding and evaluation ofregistrations is not ideal. Accordingly, a new system and method isdesired.

SUMMARY

In accordance with the teachings described herein, systems and methodsare provided for generating images for use in systems, e.g., imagingsystems. In one example, the method may include receiving at least afirst set of images, providing a first registration, providing adisplay, displaying a first image on said display, providing a userinterface, providing a second registration, and displaying a secondimage in said user interface. In one example, the system may include animage database, a display, a registration engine, wherein at least saidregistration engine comprise software instructions stored in at leastone memory device and executable by one or more processors.

The features, functions, and advantages discussed can be achievedindependently in various embodiments of the present invention or may becombined in yet other embodiments, further details of which can be seenwith reference to the following description and drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a computer device and operating environment according to oneor more aspects described herein.

FIG. 2 is an example of a software program displaying multiple images ona computer system display that may be generated by the systems andmethods described herein.

FIGS. 3A-3D are examples of software images that may be generated by thesystems and methods described herein.

FIGS. 4A-7B are examples of additional software images that may begenerated by the systems and methods described herein.

FIGS. 8A-8B are examples of software images that may be generated by thesystems and methods described herein.

FIGS. 9-11 depict flow diagrams of example methods.

DETAILED DESCRIPTION

The following detailed description is merely illustrative in nature andis not intended to limit the embodiments of the invention or theapplication and uses of such embodiments. Furthermore, there is nointention to be bound by any expressed or implied theory presented inthe preceding technical field, background, brief summary or thefollowing detailed description.

FIG. 1 illustrates an example computer device 100 and operatingenvironment 105 according to at least one aspect described herein.Computer device 100 may be in the form of a desktop computer, a laptopcomputer, a tablet computer, a server, a cellular device, a mobilephone, a mobile computer, a mobile device, a handheld device, a mediaplayer, a personal digital assistant or the like, including acombination of two or more of these items. In the illustratedembodiment, computer device 100 may include one or more software and/orhardware components, including processor 110, image database/memory 115,input-output (I/O) interface 120, an optional touch sensitive interface125, keyboard and/or mouse 130, network interface 135, optional wirelessinterface 140, audio and/or visual interface 145, user interface module150, and imaging software 155. In another embodiment, the computerdevice includes one or more of these components to form one of thecomputer devices discussed above. In yet another embodiment, thecomputer device includes one or more of these components in addition toother components. In another embodiment, the computer device may includemore or fewer components than shown or have a different configuration ofcomponents. For example, the computer device may have two or more ofeach of these components, e.g., two or more processors, imagedatabase/memories, I/O interfaces, and/or user interface modules.Although user interface module 150 is illustrated as a separatecomponent in computer device 100, in another embodiment the userinterface module 150 may be part of one or more other components, e.g.,image database/memory 115. The components illustrated in FIG. 1 may beimplemented in hardware, software or a combination of both hardware andsoftware.

The imaging software 155 may include at least one image rendering engine155A, at least one image deformation engine 155B, at least oneregistration engine 155C or algorithm, at least one user input engine155D, and at least one user confidence engine 155E. In anotherembodiment, the imaging software may include at least one registrationevaluation engine. It should be understood that the at least one imagedeformation engine, the at least one image rendering engine, and the atleast one registration engine or algorithm, at least one user inputengine, and at least one user confidence engine as described herein, maybe implemented by software instructions executing on one or moreprocessing devices. In another embodiment, however, one or moreoperations of these software engines may instead be performed by otherknown mechanisms such as firmware or even appropriately designedhardware. The image database/memory 115, as described herein, may beimplemented using one or more memory devices. For instance, in oneexample the image database may be implemented within a memory devicethat contains another database and/or the like, and in another examplethe image database may be implanted on a stand-alone or separate memorydevice. As discussed below in greater detail, the medical images areloaded into the image database/memory 115 for computing registrationsand for quality evaluation that can be used by the user to more easilyevaluate registrations that are often very complex for a user tounderstand. The plurality of medical images may include a set oftwo-dimensional (2D) slices that are received, for example, from a CTscanner or other system for capturing three-dimensional (3D) medicalimages, such that the set of 2D slices together represent a 3D medicalimage. In other examples, the plurality of medical images slices couldbe virtual, such as sagittal, coronal, or axial images (or any otherslicing angle through the image data). In another embodiment, theplurality of images may be used for two dimensional analysis.

In the illustrated embodiment, operating environment 105 may includenetwork 160, gateway 165, internet 170, and/or server 175. Operatingenvironment may include any type and/or number of networks, includingwired or wireless internet, cellular network, satellite network, localarea network, wide area network, public telephone network, cloudnetwork, and/or the like. In another embodiment, the operatingenvironment operates locally on the computer device. In the illustratedembodiment, computer device 100 may communicate with operatingenvironment 105 through server 175 by a wireless network connectionand/or a wired network connection. Further, server 175 may connectcomputer device 100 to the public telephone network to enable telephonefunctionality (voice and data) of the computer device 100.

A computer device 100 and operating environment 105 illustrate onepossible hardware configuration to support the systems and methodsdescribed herein, including at least the methods 900-1200 discussedbelow. In order to provide additional context for various aspects of thepresent invention, the following discussion is Intended to provide abrief, general description of a suitable computing environment in whichthe various aspects of the present invention may be implemented. Thoseskilled in the art will recognize that the invention also may beimplemented in combination with other program modules and/or as acombination of hardware and software. Generally, program modules includeroutines, programs, components, data structures, etc., that performparticular tasks or implement particular abstract data types.

Moreover, those skilled in the art will appreciate that the inventivemethods may be practiced with other computer system configurations,including single-processor or multiprocessor computer systems,minicomputers, mainframe computers, as well as personal computers,hand-held computing devices, microprocessor-based or programmableconsumer electronics, and the like, each of which may be operativelycoupled to one or more associated devices. The illustrated aspects ofthe invention may also be practiced in distributed computingenvironments where certain tasks are performed by remote processingdevices that are linked through a communications network. In adistributed computing environment, program modules may be located inboth local and remote memory storage devices.

The computer device 100 can utilize an exemplary environment forimplementing various aspects of the invention including a computer,wherein the computer includes a processing unit, a system memory and asystem bus. The system bus couples system components including, but notlimited to the system memory to the processing unit. The processing unitmay be any of various commercially available processors. Dualmicroprocessors and other multi-processor architectures also can beemployed as the processing unit.

The system bus can be any of several types of bus structure including amemory bus or memory controller, a peripheral bus and a local bus usingany of a variety of commercially available bus architectures. The systemmemory can include read only memory (ROM) and random access memory (RAM)or any memory known by one skilled in the art. A basic input/outputsystem (BIOS), containing the basic routines that help to transferinformation between elements within the computer device 100, such asduring start-up, is stored in the ROM.

The computer device 100 can further include a hard disk drive, amagnetic disk drive, e.g., to read from or write to a removable disk,and an optical disk drive, e.g., for reading a CD-ROM disk or to readfrom or write to other optical media. The computer device 100 caninclude at least some form of computer readable media. Computer readablemedia can be any available media that can be accessed by the computerdevice. By way of example, and not limitation, computer readable mediamay comprise computer storage media and communication media. Computerstorage media includes volatile and nonvolatile, removable andnon-removable media implemented in any method or technology for storageof information such as computer readable instructions, data structures,program modules or other data. Computer storage media includes, but isnot limited to, RAM, ROM, EEPROM, flash memory or other memorytechnology, CD-ROM, digital versatile disks (DVD) or other magneticstorage devices, or any other medium which can be used to store thedesired information and which can be accessed by the computer device100.

Communication media typically embodies computer readable instructions,data structures, program modules or other data in a modulated datasignal such as a carrier wave or other transport mechanism and includesany information delivery media. The term “modulated data signal” means asignal that has one or more of its characteristics set or changed insuch a manner as to encode information in the signal. By way of example,and not limitation, communication media includes wired media such as awired network or direct-wired connection, and wireless media such asacoustic, RF, infrared and other wireless media. Combinations of any ofthe above should also be included within the scope of computer readablemedia.

A number of program modules may be stored in the drives and RAM,including an operating system, one or more application programs, otherprogram modules, and program data. The operating system in the computerdevice 100 can be any of a number of commercially available operatingsystems and/or web client systems.

In addition, a user may enter commands and information into the computerdevice through a touch screen and/or keyboard and a pointing device,such as a mouse. Other input devices may include a microphone, an IRremote control, a track ball, a pen input device, a joystick, a gamepad, a digitizing tablet, a satellite dish, a scanner, or the like.These and other input devices are often connected to the processing unitthrough a serial port interface that is coupled to the system bus, butmay be connected by other interfaces, such as a parallel port, a gameport, a universal serial bus (“USB”), an SR interface, and/or variouswireless technologies. A monitor or other type of display device, mayalso be connected to the system bus via an interface, such as a videoadapter. Visual output may also be accomplished through a remote displaynetwork protocol such as Remote Desktop Protocol, VNC, X-Window System,etc. In addition to visual output, a computer typically includes otherperipheral output devices, such as speakers, printers, etc.

A display can be employed with the computer device 100 to present datathat is electronically received from the processing unit. In addition tothe descriptions provided elsewhere, for example, the display can be anLPD, LCD, plasma, CRT, etc. monitor that presents data electronically.The display may be integrated with computer device 100 and/or may be astand-alone display. Alternatively or in addition, the display canpresent received data in a hard copy format such as a printer,facsimile, plotter etc. The display can present data in any color andcan receive data from the computer device 100 via any wireless or hardwire protocol and/or standard.

The computer device can operate in a networked environment using logicaland/or physical connections to one or more remote computers/devices,such as a remote computer(s). The remote computer(s)/device(s) can be aworkstation, a server computer, a router, a personal computer,microprocessor based entertainment appliance, a peer device or othercommon network node, and typically includes many or all of the elementsdescribed relative to the computer. The logical connections depictedinclude a local area network (LAN) and a wide area network (WAN). Suchnetworking environments are commonplace in offices, enterprise-widecomputer networks, intranets and the Internet.

When used in a LAN networking environment, the computer device isconnected to the local network through a network interface or adapter.When used in a WAN networking environment, the computer device typicallyincludes a modem, or is connected to a communications server on the LAN,or has other means for establishing communications over the WAN, such asthe Internet. In a networked environment, program modules depictedrelative to the computer, or portions thereof, may be stored in theremote memory storage device. It will be appreciated that networkconnections described herein are exemplary and other means ofestablishing a communications link between the computers may be used.

FIG. 2 illustrates a series of images 200A-C, e.g., medical imagesand/or other images, that may be generated and/or displayed by thesystems and methods described herein. In another embodiment, othercategories of images may be used in the system described herein, e.g.,satellite imaging, photographs, microscopy, and the like. Image 200A isa coronal view, image 200B is a sagittal view, and image 200C is anaxial or transaxial view of a portion of a patient. Images 200A-C mayinclude portions of a number of muscles, bones, fat, organs, othertissues, implants, other elements, and the like. As discussed above, anyimage or combination of images in two, three, or more dimensions may beused and/or displayed in the systems and methods described herein. Inanother embodiment, less than or more than three images may bedisplayed. For purposes of the description below, a single image, twoimages, or more may be discussed and presented in the figures todescribe exemplary embodiments and features of the systems and methodsdescribed herein. However, one skilled in the art will appreciate thatany number of images and dimensionality of imaging may be included inthe exemplary embodiments and features discussed herein.

Further in FIG. 2, medical image 200B includes a first image 202 and aspyglass 204. A spyglass is defined as an interface overlay, any otherview, or any other method of displaying images that may be displayed inconjunction with at least one other image, e.g., a first image. Inanother embodiment, the spyglass may be displayed alone. The first image202 is a primary image, however, in another embodiment the first imagemay be a secondary or any other type of image. In another embodiment,the first image 202 could be a graphical representation of anatomicalboundaries or segmented regions or any other abstraction of an image ora combination of at least one such abstraction and an image. In anotherembodiment, the overlay or spyglass could be the same size as or largerthan the whole display, therefore, the first image may not be visible.In the illustrated embodiment of FIG. 2, overlay 204 is displayed as asquare shaped user interface, however, in another embodiment the overlaymay be configured as a user interface having another shape and/orgraphic, e.g., a circle, triangle, star, and the like. Although theoverlay is displayed as a square shaped user interface in the twodimensional image(s), e.g., image 200B, the overlay is derived from athree dimensional cube that is displayed as a two dimensional overlay orsquare in at least one image. In another embodiment, the systems andmethods described herein may include an alternative overlay or spyglass,for example a spherical or globe shape, a pyramid shape, and the like.In still another embodiment, these shapes may exist in more than threedimensions or displayed in more than two dimensions.

In the illustrated embodiment, overlay 204 is a graphical user interfacethat displays a second image 206 having a localization point 208(represented in the figure by a plus (+)) surrounded by a localizationregion or neighborhood 210. For example, the localization point 208 maybe at the center of a localization region or neighborhood 210 having a5×5×5 voxel volume or alternatively may be a 1×1×1, 2×2×2, 3×3×3, or4×4×4 voxel volume. In another embodiment, the localization region orneighborhood may be larger than a 5×5×5 voxel volume. In still anotherembodiment, the localization region or neighborhood may be a differentshape, such as a sphere, circle, or square. The overlay 204 may furtherinclude first and second axes 212 which extend from localization point208 past the perimeter of the overlay 204 to first image 202. Overlay204 is described as a user interface because a software user can use acomputer input, e.g., a mouse, touch screen, keyboard, and the like, asan input to the software to change location of the overlay 204 and thelocalization point 208, provide input and/or instructions to thesoftware, select different modes of display or interaction, and torecord qualitative comments and quantitative metrics regardingdynamically selected localization point(s) for the various embodimentimage(s) as discussed further herein. Of course, one skilled in the artof software appreciates that other user interfaces and the like may beincluded with the images discussed and illustrated herein, e.g.,companion controls, color bars or side panels that provide some of thecontrol discussed herein.

In the illustrated embodiment of FIG. 2, the second image 206 displaysthe portion of the primary that is below the overlay 204. As discussedfurther herein, the overlay may display another image, including but notlimited to a secondary image, a blended image, a registration or animage after applying a registration to the image, including but notlimited to a deformable registration and a rigid local registration, animage representing properties or aspects of a registration, an imagethat includes a checker display including at least a portion of an imageor a registration, an overlay that includes vector renderings whichindicate direction and magnitude of registration displacements, an imagethat includes a heat map display indicating a difference between two ormore registrations, an image that includes a subtraction displayindicating a difference between two or more images, or another type ofimage or overlay which displays an aspect of at least one registrationor image, an aspect of a comparison between two or more registrations,or an aspect of a comparison between two or more images.

An image processed through a complex or higher order registration, suchas deformable registration, may be difficult for a user to understandand rely on because of the difficulty of inspecting the registration forthe purpose of gaining confidence that the registration is correct or atleast portions of the registration are correct, e.g., it is difficultfor user to perceive whether a voxel in bone or other tissue in theprimary image is the same voxel in bone or other tissue in the secondaryimage. For example, a point may go virtually anywhere in a deformableregistration because it can have many degrees of freedom, making it hardfor a user to determine the quality and/or reliability of theregistration. And in fact, a single registration may be accurate in someareas and inaccurate in other areas. In some cases, these registrationsmay have more than a million degrees of freedom. For all of thesereasons, the user reviewing images processed with higher orderregistrations may not have a high level of confidence in the quality ofthe registration(s). This makes communication between one user whoreviews images and assesses an image and/or registration quality (suchas a physicist in radiation oncology department or imaging lab) andanother user who may review image results and prescribe a patientdiagnosis or treatment plan (such as a physician) especially inefficientand difficult. On the other hand, local rigid registrations have sixdegrees of freedom (3 in translation and 3 in rotation) and may bereviewed and understood, visually or otherwise, by a user with a higherlevel of confidence. Additionally, it is easier to perceive accuratepoint-by-point mapping and correlation through these simplerregistrations that display clearly understandable context around eachpoint. Therefore, as an example, in the systems and methods describedherein, the higher order registrations may be approximated by a best fitlocal rigid registration and the secondary image processed through thebest fit local rigid registration for each localization point selectedby the user. If the best fit local rigid registration is determined tobe acceptable by the user, the user will have a higher level ofconfidence that the higher order registration close to the localizationpoint has a good level of quality, e.g, the points in each image nearthe localization do indeed correlate, and the image processed with thehigher order registration may likely be relied on by the user, e.g., thephysicist and/or the physician. On the other hand, if the best fit localrigid registration is determined to be unacceptable to the user, theuser may record this as a bad result, rerun the registration in order toimprove the local area, or reject the result in this area.

FIGS. 3A-3D illustrate a series of images 300A-D that may be generatedand/or displayed by the systems and methods described herein. Similar toimage 200B illustrated in FIG. 2, images 300A-D illustrate a spyglass304A-D that is an interface overlay that is displayed in conjunctionwith a first image 302. Further, each overlay 304A-D in FIGS. 3A-3Ddisplays a second image 306A-D, respectively, inside the overlay, andalso includes a localization point 308 designating a center of thelocalization region or neighborhood 310. As discussed further herein, auser can dynamically select at least one localization point with acomputer input device, e.g., a computer mouse, touch screen and thelike.

In FIG. 3A, the image registration engine (discussed above) calculates afirst registration, e.g., a deformable registration, however, the secondimage 306A displays an image processed by a second registration, e.g. abest fit local rigid registration, based on the localization point 308selected by the user. In one embodiment, the higher order, e.g.deformable, registration is approximated by the best fit local rigidregistration. As discussed above, a user viewing the second image 306Acan better evaluate the best fit local rigid registration on a point bypoint basis dynamically, e.g., continuously on the fly, and determine ifthe best fit local registration, a surrogate for the first registration,is a sufficiently accurate registration in this area, and morespecifically, whether the voxels at the localization point of thesecondary and primary images correspond. For example, a user may reviewthe second image 306A close to the localization point 308 and determinethat the best fit local rigid registration has an acceptable quality.Further review of the second image closer to the square perimeter of thelocalization region or neighborhood 310 or even the spyglass 304A mayillustrate that some of these locations have unacceptable quality, e.g.,the local rigid registration may not acceptably align the source imagewith the target image. As discussed below, these peripheral mismatchesmay be in fact acceptable if the user moves the localization (point) tothe previously designated unacceptable quality location and the new bestfit local rigid registration displays an image having an acceptablequality registration. For example, in FIG. 3A, first region 314A(designated by a circle) of second image 306A may be determined by auser to have an acceptable/good quality and second region 316A(designated by a circle) may be determined to have unacceptable/poorquality. However, after the user moves the localization point 308 closeto the second region 316A, the image registration engine/algorithm willrerun (for every move of the localization point) and the user will againdetermine whether the updated best lit local rigid registrationdisplayed in the second image 306A has an acceptable quality. Asdiscussed below, this checking of other locations can be rerun as manytimes as the user requires to satisfy a quality requirement. Alternativeexemplary systems and methods are discussed further below.

FIG. 3B is similar to FIG. 3A, except image 300B includes a second image306B of a heat map in overlay or spyglass 304B. In this embodiment, theimage system has access to a primary image, a first registration, and anumber of best fit local rigid registrations). The heat map is agraphical representation shown in colors, black and white, crosshatching, another graphical representation, or the like that representsthe difference between the first registration and the best fit localrigid registration, e.g., the difference in magnitude and/or directionbetween the registration vectors for each pixel/voxel in the firstregistration, e.g., a deformable registration, and the best fit localrigid registration. First region 318B (dark cross hatching) may be aregion in the second image 306B where the difference between theregistrations is small and second region 320B (light cross hatching) maybe a region where the difference between the registrations is large. Inthe illustrated embodiment, the different styles of cross hatchingrepresent different levels or groupings of differences between theregistrations, for example the darkest cross hatching may representdifferences between registration with a magnitude between 0 and 1 mm. Asdiscussed above and further below, a user may elect to investigatewhether the differences resolve, i.e, approach 0 in magnitude, by movingthe localization point over the areas with larger differences todetermine if the differences between the registrations resolve. If thedifferences resolve (less difference between the registrations asindicated by the heat map), then the quality of the deformableregistration may be determined by the user as acceptable. If the userdetermines that the differences do not resolve, the user may elect torun another registration and to conduct another quality review or recordareas which may not have registered accurately as described herein.Additionally, for rigid structures, such as bony anatomy, a user mayelect to localize to one area in the bone and review remote areas of thesame bone to determine whether the best fit local rigid registrationfrom the first area is a good approximation also for the second area. Inthis case, the user may determine that the rigid bone has beenregistered rigidly which may generally be deemed desirable and accurate.

FIG. 3C is similar to FIG. 3A, except image 300C includes a second image306C of a checkered overlay or spyglass 304C. In the illustratedembodiment, the checkered overlay or spyglass 304C includes fourquadrants, including top left 320C and bottom right 322C and top right324C and bottom left 326C. Top left 320C and bottom right 322C quadrantsdisplay an image substantially similar to a portion of the second imagedisplayed in FIG. 3A, therefore, the top left and bottom right quadrantsare displaying portions of an image processed by a second registrationor a best fit local rigid registration based on the localization point308 selected by the user. In the illustrated embodiment, top right 324Cand bottom left 326C are displaying the primary image of first image302. In another embodiment, one quadrant, three quadrants, or a numberof quadrants less than a total number of quadrants display portions ofan image processed by a second registration or a best fit local rigidregistration. In yet another embodiment, the checkered overlay includesmore than 2×2 quadrants, e.g., 2×3, 3×3, etc. Generally, the user wouldinspect the intersection between the quadrants to determine whether thetwo different images matched at these intersection lines. Wherever thereis discontinuity, the registration may be deemed inaccurate, and wherethere are smooth continuations of anatomy from one image to the otheracross these quadrants, the registration may be deemed acceptable.

FIG. 3D is similar to FIG. 3A, except image 300D includes a second image306D that illustrates a subtraction graphic in the overlay or spyglass304D. In this embodiment, the system uses at least the primary image andsecondary image processed through the second registration, e.g. the bestfit local rigid registration, to determine a subtraction value for eachimage pixel or voxel, which is the difference between the primary andsecondary image units at each pixel or voxel. The subtraction value isgraphically represented in the second image 306D in colors, crosshatching, and/or the like. For example, first region 326D may be aregion in the second image 306D where the subtraction absolute value islarge, because, for example, the primary and secondary images are notaccurately registered, and second region 328D may be considered a regionwhere the subtraction absolute value is small, because, for example, theprimary and secondary images are accurately registered. In theillustrated embodiment, the distinct styles of cross hatching representdifferent levels or groupings of subtraction values between the images,e.g., the clear or white portions of the image may represent where theabsolute difference between the two images is less than 100 Hounsfieldunits on a Computed Tomography (CT) medical image. In anotherembodiment, another number and set of styles can be used to representthe levels. As discussed herein, a user may move the localization pointover the image areas with larger differences to determine if thedifferences between the images may resolve, i.e. approach a subtractionvalue of 0, potentially indicating an accurate registration in theseareas.

FIGS. 4A-4B illustrate a series of images 400A-B that may be generatedand/or displayed by the systems and methods described herein. Similar toimage 300A illustrated in FIG. 3A, images 400A-B illustrate a spyglass404A-B that is an interface overlay that is displayed in conjunctionwith a first image 402. Further, each overlay 404A-B in FIGS. 4A-4Bdisplays a second image 406A-B inside the overlay. Specifically, thesecond images 406A-B display a 50-50 blended image or a display of apercentage of the secondary image (50%) after applying the secondregistration (rigid local registration) blended with a percentage of theprimary image (50%). In another embodiment, the overlay 404A-B maydisplay a blend selected by the user, the blend being adjustable between0-100% of the source applied to a registration over a complimentary0-100% of the target image. Overlapping features in the blended imageallow the user to determine the quality of the registration from pointto point in a local area. FIGS. 4A-4B further illustrate how a user candynamically move from one localization (point) to another. In theillustrated embodiment, the software system calculates and displaysadditional blended images based on additional rigid local registrationsas a user moves the localization point 408 from a first region 412A ofimage 400A to a second region 414B of image 400B. In this embodiment,movement of the localization point 408 illustrates how a user mayinvestigate whether the overlapping features represented in first area416A of image 400A (area represented as a circle) improve by moving thelocalization point 408 up the sagittal image to the second area 418 ofimage 400A which corresponds to area 416B illustrated in image 400B(area also represented as a circle). As discussed herein, a user candynamically move to as many localization points as needed, and eachmovement produces another rigid local registration that can be viewed bythe user. As discussed herein, each registration or a portion of eachregistration may be marked good, poor, etc. to communicate to otherusers and/or to provide an input to a rerun a registration. Indetermining whether the registration is accurate, a user may look forsharp features in the overlapping region which appear to correspond wellbetween the two images. In areas where sharp features are absent, a usermay look for smooth transitions in the best fit local rigid registrationwhen traversing localizations between sharp features, indicating asmooth first registration which may be deemed to be desirable in theselow contrast areas of the images. In another embodiment, a localregistration metric may be computed in order to score the localregistration in this area and optionally presented to the user throughthe user interface.

FIGS. 5A-5B illustrate a series of images 500A-B that may be generatedand/or displayed by the systems and methods described herein. Similar toimage 300B illustrated in FIG. 3B, images 500A-B illustrate a spyglass504A-B that is an interface overlay that is displayed in conjunctionwith a first image 502. Further, each overlay 504A-B in FIGS. 5A-5Bdisplays a second image 506A-B inside the overlay. Specifically, thesecond images 506A-B display a heat map and a primary image compositeblended as described in reference to blending in FIGS. 4A-4B. Asdiscussed above in reference to FIG. 3B, the heat maps displayed inFIGS. 5A-B axe graphical representations shown in colors, black andwhite, cross hatching, another graphical representation, or the likethat represents the difference between the first registration and thesecond registration (e.g. best fit local rigid registration), e.g., thedifference between the registration vectors for each pixel/voxel in thefirst registration and the second registration. Substantially similar tothe example illustrated and described in FIGS. 4A-4B, a user candynamically move the localization point 508 from a first region 512A ofimage 500A to a second region 514B represented in image 500B.Specifically, the images illustrate a user moving between a firstlocalization point or area 516A in FIG. 5A to a second localizationpoint or area 516B in FIG. 5B where a user determines if the differencesshown in the heat map resolve.

FIGS. 6A-6B illustrate a series of images 600A-B that may be generatedand/or displayed by the systems and methods described herein. Similar toimage 300C illustrated in FIG. 3B, images 600A-B illustrate a spyglass604A-B that is an interface overlay that is displayed in conjunctionwith a first image 602 which is checkered. The checkered overlay orspyglass 604A-B includes four quadrants, including top left 620A-B andbottom right 622A-B and top right 624A-B and bottom left 626A-B. Topleft 620A-B and bottom right 622A-B quadrants display an imagesubstantially similar to a portion of the second image displayed in FIG.4A; therefore, the top right 624A-B and bottom left 626A-B quadrants aredisplaying the primary image or first image 602. Similar to FIGS. 4A-4Band 5A-5B, a user can dynamically move the localization point 608 toresolve image registration issues as discussed above.

FIGS. 7A-7B illustrate a series of images 700A-B that may be generatedand/or displayed by the systems and methods described herein. Similar toimage 300D illustrated in FIG. 3D, images 700A-B illustrate a spyglass704A-B that is an interface overlay that is displayed in conjunctionwith a first image 702 having a subtraction graphic. As discussed above,the system uses at least the primary image and secondary image processedthrough the second registration, e.g. the best fit local rigidregistration, to determine a subtraction value for each image pixel orvoxel, as described above in reference to FIG. 3D. The subtraction valueis graphically represented in second image 306D in colors, crosshatching, and/or the like. Specifically, the spyglass 704A-B displays asubtraction and a primary image composite blended as described inreference to blending in FIGS. 4A-4B. Similar to FIGS. 4A-4B, 5A-5B, and6A-6B, a user can dynamically move the localization point 708 to attemptto resolve differences between the images as discussed herein.

In another embodiment, multiple display modes may be further blendedtogether in the spyglass. For example, the primary and secondary blendas shown in FIGS. 4A-B may be further blended with the heat map as shownin FIG. 3B.

FIGS. 8A-8B illustrate a series of images 800A-B that may be generatedand/or displayed by the systems and methods described herein. Image 800Adisplays a spyglass 804A-B that is an interface overlay that isdisplayed in conjunction with a first image 802 having a heat map andprimary image composite 806A, for example. During review of the image806A, a user can enter a quality rating on image 806A. For example, auser can click the right mouse button for localization points that havegood quality and the left mouse button for localization points that havelow quality. In other embodiments, other input mechanisms and controlscan be used to track/record these user quality determinations. Forexample, a toggle system using a combination of keyboard or touch screenbuttons (for example) may allow a user to toggle between recording modes(e.g. “good”, “bad”, and/or other notations or classifications, and/orinstructions concerning what types of improvements are required toimprove the registration in the local area under review), in order torecord classifications of the local registration during registrationreview. During this interaction, the software and/or system tracksand/or records these inputs or modes and highlights/shades sections inthe registration record, as illustrated in FIG. 8B. For example, in theillustrated embodiment, first area 812 may be designated as good qualityand second areas 814A-B may be designated as poor quality. Theregistration algorithms may then take these or other designations asinput(s) to produce another registration. For example, a registrationalgorithm, when executed with this input and possibly the additionalinput of the previous registration, may seek to maintain registrationparameters from the previous registration in areas marked as “good” andseek to replace registration parameters from the previous registrationin areas marked as “bad”. In another embodiment, in the registrationalgorithm the registration parameters in “good” areas may be used todirectly influence or constrain the registration parameters in the “bad”areas. In another embodiment, a user may use a touch screen to designategood and poor quality locations. In still another embodiment, the usermay elect to display other spyglass modes during this recording process,such as blended overlay, subtraction, or checkered overlay. In yetanother embodiment, the designation of the quality markings may beautomated based on a registration rating module or engine that mayconsider input from a heat map, checkered overlay, and/or subtractionmodules or engines.

FIG. 9 is a flow diagram depicting an example method 900 for computingand/or evaluating a registration. Method 900 may be used in a system,software program, computer device, operating system, or any combinationof the same. The first set of images are received or provided at 905 anda second set of images are received or provided at 910. For example, thefirst set of images may include the primary images and the second set ofimages may include the secondary images. In another embodiment, thesecond set of images may be optional. For example, if the firstregistration at 915 is loaded from another source and only registrationoverlays are displayed, e.g. the heat map, a second image may not berequired.

Using the first set of images and the second set of images, at least afirst registration algorithm runs at 915 and provides the at least firstregistration at 920. For example, the at least first registration may bea deformable registration. In another embodiment, a first registrationmay be loaded from another source, e.g., another memory location inanother system. In yet another embodiment, the at least firstregistration is not a deformable registration. For example, the at leastfirst registration could be a rigid registration. At least a first imageis displayed at 925, at least one user interface overlay or spyglass isprovided on at least a portion of the displayed image at 930, and alocalization point or position is defined or selected by a user in realtime at 935. At least a second registration algorithm is optionally runand at least a second registration is provided at 940. For example, atleast a second registration algorithm may approximate the firstregistration with a best fit lower order registration; e.g., the atleast a second registration may be a best fit local rigid registration.In another embodiment, the at feast second registration algorithm mayapproximate the first registration with a global rigid registration. Asdiscussed above, the purpose of providing a second registration or abest fit local rigid registration is to allow the user to see at leastone lower order rigid registration to help the user understand the firstregistration or the complex, higher order deformable registration. Inanother embodiment, the at least one second registration might be aregistration independently computed by optimizing similarity between theprimary and secondary images in a local region rather than a best fitregistration which is an approximation to the first registration. Thisat least second registration may be a rigid registration or may be ahigher order registration. In yet another embodiment, the firstregistration may be a lower order registration and the at least secondregistration may be a higher order registration computed using analgorithm to optimize the image matching between the primary andsecondary images. This embodiment would enable the user to determine thequality of the lower order registration in comparison with the higherorder registration which may be more accurate at very local imagefitting, but may have other negative trade-offs which make it lessdesirable, such as inaccuracies in certain parts of the registration. Inthese embodiments, the purpose of the at least second registration is asa general comparison to the first registration as opposed to someembodiments where the at least second registration is used to betterunderstand the first registration. In addition, both registrations couldbe scored based on their quality or other properties and this recordcould be fed back into a registration algorithm to regenerate a newregistration.

At 945, at least a portion, an aspect, or a property of at least oneregistration or at least one image is displayed over the first image inthe user interface overlay (spyglass) about the at least onelocalization point of the overlay. For example, the second imageprocessed through second registration may be displayed in the overlayover a first image that contains a primary image (e.g., FIG. 3A). Inanother embodiment, the heat map described and illustrated in FIG. 3B,the checkered overlay described and illustrated in FIG. 3C, or thesubtraction value graphic described and illustrated in FIG. 3D isdisplayed in the overlay over at least a portion of the first image. Inanother embodiment, another aspect or property about the firstregistration, the second registration, or the comparison between the atleast two registrations is displayed in the overlay over at least aportion of the first image, e.g. vectors displaying the magnitude anddirection of the registration(s) at each pixel or voxel or a differenttype of heat map which depicts a metric of the local properties of theregistration, including but not limited to compressibility or anaccuracy metric.

A user may redefine the localization point or position at 950,triggering the dynamic and user controlled rerunning of at least asecond registration and providing at least a second registration at 940,which is then displayed over the first image in the user interfaceoverlay (spyglass) at 945. Every time the user redefines thelocalization point or position at 950, the system goes through anotherpass at 950, 940, and 945. In another embodiment, the system may beconfigured (programmed) to include an automated process that triggersfrom 945 back to 940 to rerun the at least a second registration toprovide at least a second registration at 940 (e.g. a new or updatedbest fit local rigid registration) that is then displayed over the firstimage in the user interface overlay (spyglass) at 945 to help the userevaluate the registration. At 955, user notations in the user interlaceoverlay or spyglass based on a bad or negative quality assurance or someother review triggers a partial rerun of the method starting at 915where at least a first registration algorithm is reran, possibly basedon these user inputs, or loaded from another system as discussed above.Alternatively, at 955, the user notations could trigger and be used asinputs for a rerun of the method starting at 940 where the at least asecond registration algorithm is rerun, possibly based on these userinputs, for example to bias the registration away from had results andto results more acceptable to the user. Alternatively, after 955 thesystem may come to an end as illustrated in FIG. 9. In anotherembodiment, a software menu item or another input may trigger a partialrerun of the method starting at 915.

FIG. 10 illustrates another flow diagram depicting an example method1000 for computing and evaluating a registration. Method 1000 issubstantially similar to method 900 discussed above, except method 1000includes additional quality assurance (“QA”) components at 1055, 1060,and 1065. In other words, 1005, 1010, 1015, 1020, 1025, 1030, 1035,1040, 1045, and 1050 are substantially similar to 905, 910, 915, 920,925, 930, 935, 940, 945, and 950, respectively.

At 1045, at least a portion, an aspect, or a property of at least oneregistration or at least one image is displayed over the first image inthe user interface overlay (spyglass) about the at least onelocalization point of the overlay, similar to 945 in FIG. 9. Similar tothe method 900 discussed above, the system may be configured(programmed) to include an automated process that triggers from 1045back to 1040 to rerun the at least a second registration to provide atleast a second registration at 1040 that is then displayed over thefirst image in the user interface overlay (spyglass) at 1045 to assistthe user in evaluating the registration. In addition, method 1000includes at 1055 recording of user confidence for at least onelocalization point. User confidence can be recorded at 1055 as a videorecording or “bird's eye” view of a user review of the overlay for atleast one localization point, e.g., localization point by localizationpoint. This record is a method whereby communication can be improvedbetween personnel responsible for review and quality assurance of aregistration (e.g., a physicist) and personnel responsible fordecision-making regarding a registration (e.g., a physician).Alternatively, user confidence at 1055 can be entered on the overlay inthe form of highlighting or drawing a perimeter about a portion of theimage displayed in the overlay, recorded marks (positive and negativeindicator's entered by the user through a mouse or another computerdevice input), and painting of at least a portion of a registration inthe overlay or on the first image. In another embodiment, notationsconcerning the registration other than user confidence can also berecorded, e.g. whether a certain area should be considered a rigidstructure, which might be used as an input into a registration algorithmat 1015 or 1040. In yet another embodiment, the user may interact withthis registration spyglass in other ways, such as to contour (i.e.,segment or outline) a structure of interest or a portion of a structureof interest based on the image(s) which are displayed, potentiallydynamically, in the spyglass. For example, a local rigid registrationcould be interactively updated by changing the localization point andimmediately used to outline a close-by and accurately registered edge ofa structure in one of the images. This process could be continued untilthe entire structure of interest is contoured. In another embodiment,these local registrations could be composited together using aregistration approximation algorithm into a single global registration,optionally considering as in input user recorded user notations abouteach local registration. In another embodiment, method 1000 can becompleted automatically, therefore, the entire image is traversed tocompute local registrations in order to composite a deformableregistration.

At 1060, a report or metric of at least one localization point isproduced that provides a qualitative and/or quantitative evaluation ofthe registration about local portion(s) of the image(s). In oneembodiment, the report contains a series of captured images of reviewedlocalization points, each with an indication of how the registration(s)reviewed were classified by the user, e.g., as good or bad. In anotherembodiment, the report is at least partially comprised by a new volumethat indicates the classifications which were recorded. This could berepresented as a set of images similar to FIGS. 8A-B. In anotherembodiment, a table summarizing the recorded user classifications couldbe included in a report. In another embodiment, the report could besaved and loaded, including the new volume with regional classificationsrecorded, to be interacted with in a software system by another user,e.g., the report may be saved by a physicist and loaded and reviewed bya physician. At 1065, the method optionally reruns a registration basedon user input, including, for example, the classifications describedabove, at 1060 and continues with the method at 1015, 1020, and thelike.

FIG. 11 illustrates yet another flow diagram depicting an example method1100 for computing and evaluating a registration. Similar to method 900illustrated in FIG. 9, method 1100 includes a first set of imagesreceived or provided at 1105 and a second set of images received orprovided at 1110. Using the first set of images and the second set ofimages, at least a first registration algorithm runs at 1115. In anotherembodiment, a first registration may be loaded from another source,e.g., another memory location in another system. At 1120, an automatedprocess controls the computation of registrations using localizationpoints of a number of predetermined or user adjustable pixel or voxellocations. At 1125, the automated process approximates a best fit localregistration for the first registration at each localization point. Inanother embodiment a second registration algorithm is executed takingthe first and second set of images and optimizing their alignment to oneanother. From 1125, the method may include combining local registrationsinto a new global registration (1130) or combining differences betweenfirst and second registration into a new image (1135). In still anotherembodiment, a new image or volume is generated where each pixel or voxelhas the value of the difference between the first registration and thesecond, local registration. This generates a volume analogous to theheat map previously described. In another embodiment, a new registrationis generated which is a combination of each of the local computedregistrations.

The embodiments of this invention shown in the drawing and describedabove are exemplary of numerous embodiments that may be made within thescope of the appended claims. It is understood that numerous otherconfigurations of the graphical user interfaces may be created takingadvantage of the disclosed approach. In short, it is the applicant'sintention that the scope of the patent issuing herefrom will be limitedonly by the scope of the appended claims.

What is claimed is:
 1. A method, comprising: utilizing a firstregistration between a source image and a target image to generate asecond registration between the source image and the target image;displaying the target image on a display device; receiving user inputindicative of at least a portion of the target image on which to displayan overlay image, wherein the second registration is generated inresponse to the user input; and displaying, on the display device, theoverlay image over the portion of the target image, the overlay image isgenerated based at least in part on the second registration between thetarget image and the source image.
 2. The method of claim 1, wherein theoverlay image comprises a portion of the source image as transformedaccording to the second registration.
 3. The method of claim 1, whereinthe overlay image indicates an accuracy of the first registrationutilized to generate the second registration.
 4. The method of claim 1,wherein utilizing the first registration to generate the secondregistration further comprises approximating the first registration togenerate the second registration.
 5. The method of claim 1, whereinutilizing the first registration to generate the second registrationfurther comprising generating a best-fit local registration between thesource image and target image with respect to a point of at least one ofthe source image or the target image.
 6. The method of claim 1, furthercomprising: receiving user input indicative of a change to the portionof the target image on which to display the overlay image; regeneratingthe second registration responsive to a change of the localizationpoint; and updating display of the overlay image based on theregenerating of the second registration.
 7. The method of claim 1,further comprising obtaining the first registration from a data store,wherein the first registration is pre-computed.
 8. The method of claim1, further comprising receiving user input indicative of a quality ofthe first registration with respect to an area of at least one of thesource image or the target image.
 9. The method of claim 8, furthercomprising generating a third registration between the source image andthe target image based on the user input, wherein the third registrationis based at least in part on the second registration.
 10. The method ofclaim 1, wherein the first registration is a deformable registration.11. The method of claim 1, wherein the second registration is a rigidregistration.
 12. A non-transitory, computer-readable storage mediumhaving stored thereon computer-executable instructions that, whenexecuted by a processor of a computing device having a user interfaceand a display, configure the processor to: generate a secondregistration between a source image and a target image based on a firstregistration between the source image and the target image; display thetarget image on the display; receive user input via the user interfaceindicative of a selection of a point of the target image; and display anoverlay image on at least a portion of the target image established bythe point selected, the overlay image is generated based on the secondregistration with respect to the portion of the target image on whichthe overlay image is displayed.
 13. The non-transitory,computer-readable storage medium of claim 12, wherein the overlay imageindicates at least an accuracy of the first registration, between thetarget image and the source image, relative to the portion of the targetimage on which the overlay image is displayed.
 14. The non-transitory,computer-readable storage medium of claim 12, further storinginstructions that configure the processor to dynamically update theoverlay image in response to a change of the point selected.
 15. Thenon-transitory, computer-readable storage medium of claim 12, furtherstoring instructions that configure the processor to dynamicallygenerate the second registration in response to a change of the pointselected.
 16. The non-transitory, computer-readable storage medium ofclaim 12, wherein the first registration is a deformable registrationand the non-transitory computer-readable storage medium further storesinstructions that configure the processor to generate, using a rigidregistration algorithm, the second registration as an approximation ofthe first registration.
 17. A system, comprising: a display; an inputdevice; at least one processor; and a memory storing computer-executableinstructions that, when executed by the at least one processor,configure the processor to: generate a rigid registration between asource image and a target image based on a deformable registrationbetween the source image and a target image; display the target image onthe display; receive, via the input device, user input indicative of aselection of a portion of the target image on which to display anoverlay image, wherein the rigid registration is generated as a localapproximation of the deformable registration within the portion of thetarget image selected; and display the overlay image on the portion ofthe target image, the overlay image is generated based at least in parton the rigid registration between the source image and the target image.18. The system of claim 17, wherein the overlay image comprises aportion of the source image transformed according to the rigidregistration to indicate an accuracy of the deformable registration.